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ABSTRACT 

A content analysis was conducted of three educational 
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categories: (1) descriptive; (2) bivariate correlations; (3) t-test; 

(4) nonparame t r i c statistics; (5) met a~ana lys i s ; (6) analysis of 

variance (ANOVA) and analysis of covariance (ANCOVA) ; (7) 

psychometric theory; (8) multiple correlation and regression; (9) 
multivariate analysis; (10) factor and cluster analysis; (11) LISREL 
computer program analysis; (12) Bayesian statistics; (13) simulation; 
(14) modeling; (15) qualitative methods of several types; and (16) 
graphic methods. Results are consistent with those of other studies 
in that the most commonly used methods were ANOVA ai:d ANCOVA, 
multiple regression, bivariate correlation, descriptive statistics, 
multivariate analysis, nonparamet ri c statistics, and t~tests. The 
major difference in current methodology is the increase in the use of 
qualitative methods. (Contains 5 tables, 4 graphs, and 18 
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Professors of educational statistics, measurement, 
evaluation, and research have the task of teaching methodology 
to each new generation of researchers. In this role they 
frequently must make difficult decisions about which topics are 
essential for all educational researchers and should be included 
in the doctoral tool sequence in statistics and which courses and 
topics are recommended for specialists in a quantitative doctoral 
program. Because they are expected to provide students with 
enough expertise to be intelligent consumers as well as 
producers of research, the choice of topics to cover in a limited 
number of courses becomes critical. 

One approach to evaluating the extent to which doctoral 
students are adequately prepared in educational statistics and 
research methods courses is to conduct a content analysis of 
educational research journals, and to then compare the methods 
and techniques used in these research articles to what students 
are actually learning in graduate school. In this study, the first 
part of this research will be conducted, namely a content 
analysis of three educational research journals published by the 
American Educational Research Association. Such a review of 
research techniques has appeared frequently in the educational 
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literature (Dillon, 1983; Elmore & Woehlke, 1988; Goodwin and 
Goodwin, 1985a, 1985b; Gordon, Nucci, West, Hoerr, Uguroglu, 
Vukosavich & Tsai, 1984; Shaver & Norton, 1980; Smith & 
Caulley, 1981; Walberg, Vukosavich & Tsai, 1981; Willson, 1980). 

It is apparent that periodic updates of the quantitative and 
qualitative methods used in research literature has a rich 
history. However, none has been completed since 1987 for the 
journals published by the American Educational Research 
Association. It is the intent of this paper to extend our earlier 
work to the present time by reviewing the quantitative and 
qualitative techniques used in articles published in American 
Educational Research Journal (AERJ), Educational Researcher 
(ER) and Review of Educati onal Research (RER) for the period 
1988 to 1995 and to compare those results with the findings 
from our review conducted for the period 1978 to 1987 (Elmore 
& Woehlke, 1988). 

Method 

All articles appearing in AERJ, ER, and RER for the period 
from 1988 through 1995 were selected for examination. Book 
reviews, annual meeting notices, directories, and minutes of 
meetings were omitted from the review. 

The coding process included two steps: (a) reading the 
article to identify every research method or statistical technique 
used and (b) categorizing all the methods and techniques 
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identified in each article. Where more than one method was 
employed in a single article, all methods were coded into 
appropriate categories; as a result, the total coded methods may 
exceed the total number of articles reviewed. The categories 
employed to code research methods or statistical techniques 
u^ed in the articles were: 

Descriptive: frequencies, percentages, ratios, rates, 
measures of central tendency and variability; 

Bivariate correlation: Pearson product-moment correlation 
coefficients or other coefficients used with two 
variables; 

t-test: two-group comparison of means; 

Nonparametric: statistics used with nominal or ordinal 
data; 

Meta-analysis: syntheses of research using any of three 
techniques proposed to date by Glass, McGaw and 
Smith (1981), Rosenthal and Rubin (1982) and 
Hedges and Olkin (1985); 

ANOVA/ANCOVA: hypotheses tested for group differences; 

Psychometric theory, application of statistics to the 
development of measuring instruments; 

Multiple correlation/regression: methods used to relate 
more than one independent variable to a single 
continuous dependent variable; 
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Multivariate: techniques using more than one dependent 
variable; 

Factor/ cluster: correlational techniques used to isolate 
subsets of related variables/observations; 

LISREL: analysis of covariance structures using maximum 
likelihood estimation (e. g . . path analysis, 
confirmatory factor analysis; 

Bayesian: use of Bayesian statistical methods rather than 
Neyman-Pearson; 

Simulation: analysis of simulated rather than empirical 
data; 

Modeling: empirical test of a theoretical model; 

Qualitative: use of specific techniques associated with 

educational evaluation (e.g.. naturalistic observations; 
field, ethnographic, phenomenological, and case 
studies) 

Graphic methods: use of graphic methods such as bar 

charts, line graphs, scatter diagrams, histograms, 
polygons, and box and whisker plots 

The first author coded all the ER articles and the second 
author coded all AERJ and RER articles. The authors used the 
same coding process and categorization of techniques developed 
for the 1978 to 1987 study with the addition of one new 
method, graphic methods. The authors did consult each other 
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after coding the first year of each journal concerning any 
problems encountered and they mutually agreed to add the 
“graphic techniques” category after reviewing their respective 
journals. 

Results 

The frequency of research methods or statistical 
procedures used in AERJ for each of the eight years from 1988 
to 1995, the total number of articles reviewed by year, and the 
total frequency accumulated for each method across the eight 
years are shown in Table 1. The total for the previous 10 years 
(1978 to 1987) and the accumulated total for 18 years (1978 to 
1995) for each category are reported in the first and last 
columns, respectively, for comparison. Similarly, the same 
information for ER, RER, and the three journals combined is 
contained in Tables 2, 3, and 4, respectively. Table 5 contains 
the rank order of methods used in AERJ, ER, RER, and the 
three journals combined for the three time periods (1978 to 
1987, 1988 to 1995, and 1978 to 1995). In order for a method 
to receive a rank, the method had to have a frequency of at least 
ten. 

The six most frequent methods used in AERJ in rank 
order for the time period 1978 to 1987 were ANOVA/ANCOVA, 
multiple regression/correlation, multivariate, bivariate 
correlation, nonparametric, and t-test; for the time period 1988 
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to 1995 were qualitative, ANOVA/ANCOVA, multivariate, 
bivariate correlation, multiple regression/correlation, and 
LISREL and t-test tied; and, for the 18 year time period 1978 to 
1995 were ANOVA/ANCOVA, multiple regression/correlation, 
multivariate, qualitative, bivariate correlation, and 
nonparametric. 

The most frequent methods used in ER in rank order for 
the time period 1978 to 1987 were descriptive, multiple 
regression/correlation, and bivariate correlation; for the time 
period 1988 to 1995 were descriptive and graphic methods; 
and, for the 18 year time period 1978 to 1995 were descriptive, 
graphic methods, bivariate correlation, and multiple regression/ 
correlation. Since a technique had to have a frequency of at least 
10 to be ranked, fewer than six techniques were ranked for ER. 

For RER the most frequent method used across all three 
time periods was meta-analysis. Since a technique had to have a 
frequency of at least 10 to be ranked, only one technique was 
ranked for RER. 

The six most frequent methods used in the three journals 
combined in rank order for the time period 1978 to 1987 were 
ANOVA/ANCOVA, descriptive, multiple regression/correlation, 
bivariate correlation, multivariate, and nonparametric; for the 
dme period 1988 to 1995 were descriptive, qualitative, 
ANOVA/ANCOVA, graphic methods, bivariate correlation, and 



meta-analysis; and, for the 18 year time period 1978 to 1995 
were descriptive, ANOVA/ANCOVA, multiple 
regression/correlation, bivariate correlation, multivariate, and 
qualitative. 

Discussion and Conclusions 

The results for AERJ and the three journals combined for 
all three time periods are similar to those reported by Goodwin 
and Goodwin ( 1985a) for the Journal of Educational Psycholog y 
from 1979 to 1983 in which the most frequent methods used in 
rank order were ANOVA/ANCOVA, bivariate correlation, t-test, 
multiple regression, multivariate, and nonparametric 
techniques. Similarly, our results for AERJ and the three 
journals combined are consistent with the findings of Goodwin 
and Goodwin (1985b) for AERJ from 1979 to 1983 in which the 
most frequent methods reported in rank order were 
ANOVA/ANCOVA, multiple regression, bivariate correlation, 
descriptive, multivariate, nonparametric, and t-test. The only 
major difference is the substantial increase in the use of 
qualitative methods in AERJ over the last eight years. The 
frequency for the ten-year period from 1978 to 1987 was four 
while the frequency for the most recent eight-year time period 
(1988 to 1995) was 76. 

Meta-analysis was the most frequent technique found in 
RER which is consistent with the journal's editorial policy. The 



use of meta-analysis in RER has increased over the time periods 
studied. The frequency for the ten-year period from 1978 to 

1987 was 21 while the frequency for the eight-year period from 

1988 to 1995 was 31. A report of a committee of the 
Mathematical Sciences Board of the National Research Council 
(1992) stated "quantitative research synthesis— meta-analysis— 
has gained increasing use in recent years and rightly so. Meta- 
analysis offers a powerful set of tools for extracting information 
from a body of related research" (p. 2). 

In the review of journal articles it was striking to both 
authors that many articles contained visual presentation of data 
including bar charts, line graphs, scatter diagrams, histograms, 
polygons, and box and whisker plots. Therefore, a new category, 
graphic methods, was added to the categories already used by 
Elmore and Woehlke (1988). The importance of exploratory data 
analysis (Tukey, 1977) and the understanding of graphic 
methods (Wainer, 1992a, 1992b) for the statistics curriculum 
(Tukey, 1980) are confirmed in this study in which descriptive 
and graphic methods were two of the top ranked methods for 
ER and all journals combined for 1988 to 1995. 

In a survey of all PhD programs in psychology (Aiken, West, 
Sechrest, & Reno, 1990), it was concluded that the statistical 
and methodological curriculum has advanced little in 20 years 
and that measurement has experienced a substantial decline. A 
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similar survey of graduate programs training students in 
educational statistics has been conducted by Curtis and Harwell 
(1996). Their results provide information to answer an 
important question posed for this symposium: How does 
current pedagogy in educational statistics compare with 
statistical techniques used in journals published by the American 
Educational Research Association? 




li 



References 



1 1 



Aiken, L. S., West, S. G., Sechrest, L., & Reno, R. R. (1990). 
Graduate training in statistics, methodology, and measurement 
in psychology: A survey of Ph.D. programs in North America. 
American Psychologist. 45 (6). 721-734. 

Curtis, D. A., & Harwell, M. ( 1996, April). Training 
graduate students in educational statistics: A national survey. 
Paper presented at the meeting of the American Educational 
Research Association, New York, NY. 

Dillon, J. T. (1983). The use of questions in educational 
research. Educational Researcher. 12 (91. 19-24. 

Elmore, P. B., & Woehlke, P. L. (1988). Statistical methods 
employed in American Educational Research Journal. 
Educational Researcher, and Review of Educational Research 
from 1978 to 1987. Educational Researcher. 17 (91 19-20 

Glass, G. V, McGaw, B., & Smith, L. I. (1981). Meta-analvsis 
in social research . Beverly Hills, CA: Sage. 

Goodwin, L. D., & Goodwin, W. L. (1985a). An analysis of 
statistical techniques used in the Journal of Educational 
Psychology, 1979-1983. Educational Psychologist. 20( 11. 13-21. 

Goodwin, L. D., & Goodwin, W. L. (1985b). Statistical 
techniques in AERJ articles, 1979-1983: The preparation of 




12 



graduate students to read educational research literature. 
Educational Researcher. 14 (21. 5-11. 



12 



Gordon, N. J., Nucci, L. P., West, G. K., Hoerr, W. A., 
Uguroglu, M. E., Vukosavich, P., & Tsai, S. (1984). Productivity 
and citations of educational research: Using educational 
psychology as the data base. Educational Researcher. jJ2(7), 14- 
20 . 

Hedges, L. V., & Olkin, I. (1985). Statistical methods for 
meta-analysis . New York: Academic Press. 

National Research Council (1992). Combining information: 
Statistical issues and research opportunities . Washington, DC: 
National Academy of Sciences Press. [Reprinted as Combining 
information: Statistical issues and research opportunities . 
Washington, DC: American Statistical Association, 1993.] 

Rosenthal, R., & Rubin, D. B. (1982). Comparing effect 
sizes of independent studies. Psychological Bulletin. 92. 500- 
504. 

Shaver, J. P., & Norton, R. S. (1980). Randomness and 
replication in ten years of the American Educational Research 
Journal. Educational Researcher. 9 (1). 9-15. 

Tukey, J. W. (1977). Exploratory data analysis . Reading, 
MA: Addison-Wesley. 

Tukey, J. W. (1980). We need both exploratory and 
confirmatory. The American Statistician. 34 ( 1), 23-25. 




13 



Wainer, H. (1992a). Prologue— How the following article 
came to be. Educatio n al Researcher. 21 (1). 12-13. 

Wainer, H. (1992b). Understanding graphs and tables. 
Educational Researcher. 21 (1). 14-23. 

Walberg, H. J., Vukosavich, P., & Tsai, S. (1981). Scope 
and structure of the journal literature in educational research. 
Educational Researcher. 10 (8). 11-13. 

Willson, V. L., (1980). Research techniques in AERJ 
articles: 1969 to 1978. Educational Researcher. 9 (6). 5-10. 



14 



Table 1 

Methods Used in American Educational Research Journal 



JL V^Ctl 



Method 


78-87 


88 


89 


90 


91 


92 


93 


94 


i 

95 


i 

88-95 


78-95 


Descriptive 


28 


1 


0 


2 


2 


2 


1 


2 


1 


11 


39 


Bivariate correlation 


47 


7 


1 


5 


5 


2 


2 


2 


5 


29 


76 


t-test 


42 


4 


4 


4 


3 


0 


2 


3 


2 


22 


64 


Nonparametric 


46 


0 


o 

O 


0 


4 


o 

O 


2 


5 


2 


19 


65 


Meta-analysis 


13 


0 


0 


0 


0 


0 


0 


0 


0 


0 


13 


ANOVA/ANCOVA 


137 


10 


7 


8 


11 


8 


7 


7 


6 


64 


201 


Psychometric theory 


12 


0 


0 


2 


1 


0 


0 


0 


0 


3 


15 


Multiple reg./corr. 


95 


3 


3 


2 


4 


3 


3 


2 


g 


28 


123 


Multivariate 


53 


4 


3 


4 


7 


3 


3 


2 


4 


30 


83 


Factor/ cluster 


30 


0 


1 


2 


0 


2 


3 


5 


5 


18 


48 


LISREL 


38 


4 


3 


4 


3 


4 


0 


1 


3 


22 


60 


Bayesian 


1 


0 


0 


2 


0 


0 


0 


0 


0 


2 


3 


Simulation 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


Modeling 


4 


0 


1 


1 


0 


0 


0 


1 


0 


3 




Qualitative 


4 


5 


8 


8 


11 


16 


12 


7 


9 


76 


80 


Graphic methods 


NA 


0 


0 


2 


4 


0 


3 


0 


0 


9 


18 


Total articles reviewed 


400 


28 


17 


35 


38 


37 


31 


34 


25 


245 


645 




15 



15 



Table 2 

Methods Used in Educational Reseacher 



Year 



Method 


78-87 


88 


89 


90 


91 


92 


93 


94 


95 


88-95 


78-95 


Descriptive 


84 


15 


16 


11 


11 


14 


8 


8 


6 


89 


173 


Bivariate correlation 


11 


1 


1 


0 


2 


1 


0 


1 


0 


6 


17 


t-test 


2 


0 


1 


0 


0 


0 


0 


1 


0 


2 


4 


Nonparametric 


5 


0 


0 


0 


0 


0 


0 


2 


0 


2 


7 


Meta-analysis 


4 


1 


1 


0 


0 


0 


0 


1 


0 


3 


7 


ANOVA/ANCOVA 


6 


0 


0 


0 


0 


2 


0 


0 


1 


3 


9 


Psychometric theory 


5 


0 


0 


0 


0 


1 


0 


0 


1 


2 


7 


Multiple reg. /corr. 


13 


1 


0 


0 


0 


0 


0 


1 


0 


2 


15 


Multivariate 


1 


C 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Factor/cluster 


6 


0 


0 


0 


0 


0 


0 


0 


0 


0 


6 


LISREL 


1 


0 


0 


0 


1 


0 


0 


0 


0 


1 


2 


Bayesian 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Simulation 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Modeling 


0 


1 


0 


0 


0 


0 


0 


0 


0 


1 


1 


Qualitative 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Graphic methods 


NA 


5 


9 


2 


2 


5 


3 


2 


1 


29 


29 


Total articles reviewed 


347 


33 


35 


47 


40 


45 


41 


44 


43 


328 


675 




16 



16 



Table 3 

Methods Used in Review of Educational Research 



Year 



Method 


78-87 


88 


89 


90 


91 


92 


93 


94 


95 


88-95 


78-95 


Descriptive 


5 


0 


0 


0 


0 


0 


2 


0 


0 


2 


7 


Bivariate correlation 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


2 


t-test 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Non parametric 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Meta-analysis 


21 


6 


5 


3 


2 


4 


4 


2 


5 


31 


52 


ANOVA/ANCOVA 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Psychometric theoiy 


2 


0 


0 


0 


0 


o 


0 


0 


0 


0 


2 


Multiple reg. /corr. 


0 


0 


0 


0 


0 


0 


1 


0 


0 


1 


1 


Multivariate 


0 


0 


0 


0 


0 


0 


1 


0 


0 


1 


1 


Factor /cluster 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


LISREL 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Bayesian 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Simulation 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Modeling 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Qualitative 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


2 


Graphic methods 


NA 


0 


0 


0 


0 


0 


1 


0 


0 


1 


1 


Total articles reviewed 


223 


19 


29 


24 


21 


20 


26 


16 


17 


172 


395 




17 



17 



Table 4 

Methods Used in the Three Journals Combined 



Year 



Method 


78-87 


88 


89 


90 


91 


92 


93 


94 


95 


88-95 


78-95 


Descriptive 


117 


16 


16 


13 


13 


16 


11 


10 


7 


102 


219 


Bivariate correlation 


60 


8 


2 


5 


7 


3 


2 


3 


5 


35 


95 


t-test 


44 


4 


5 


4 


3 


0 


2 


4 


2 


24 


68 


Nonparametric 


52 


0 


3 


0 


4 


3 


2 


7 


2 


21 


73 


Meta-analysis 


38 


7 


6 


3 


2 


4 


4 


3 


5 


34 


72 


ANOVA/ANCOVA 


143 


10 


7 


8 


11 


10 


7 


7 


7 


67 


210 


Psychometric theory 


19 


0 


0 


2 


1 


1 


0 


0 


1 


5 


24 


Multiple reg./corr. 


108 


4 


3 


2 


4 


3 


4 


3 


8 


31 


139 


Multivariate 


54 


4 


3 


4 


7 


3 


4 


2 


4 


31 


85 


Factor/ cluster 


36 


0 


1 


2 


0 


2 


3 


5 


5 


18 


54 


LISREL 


39 


4 


3 


4 


4 


4 


0 


1 


3 


23 


62 


Bayesian 


1 


0 


0 


2 


0 


0 


0 


0 


0 


2 


3 


Simulation 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


Modeling 


4 


1 


1 


1 


0 


0 


0 


1 


0 


4 


8 


Qualitative 


6 


5 


8 


8 


11 


16 


12 


7 


9 


76 


82 


Graphic methods 


NA 


5 


9 


4 


6 


5 


7 


2 


1 


39 


39 


Total articles reviewed 


970 


80 


81 


106 


99 


102 


98 


94 


85 


745 


1715 




18 




18 



Table 5 

Rank Order of Methods Used in American Educational Research Journal, Educational 
Researcher, Review of Educational Research, and the Three Journals Combined 



Journal and Time Period 



Method 


AERJ 


AERJ 


AERJ 


ER 


ER 


ER 


RER 


RER 


RER 


Total 


Total 


Total 




78-87 


88-95 


78-95 


78-87 


88-95 


78-95 


78-87 


88-95 


78-95 


78-87 


88-95 


78-95 


Descriptive 








i 


1 


1 








2 


1 


1 


Bivariate correlation 


4 


4 


5 


3 




3 








4 


5 


4 


t-test 


6 


6.5 






















Nonparametric 


5 




6 














6 






Meta -analysis 














1 


1 


1 




6 




ANOVA/ANCOVA 


1 


2 


1 














1 


3 


2 


Psychometric theory 


























Multiple reg./corr. 


2 


5 


2 


2 




4 








3 




3 


Multivariate 


3 


3 


3 














5 




5 


Factor/ cluster 


























LISREL 




6.5 






















Bayesian 


























Simulation 
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Qualitative 
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2 
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Graphic methods 
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2 


2 


NA 






NA 
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Note: Only the top six qualitative/ quantitative techniques were ranked for each journal for 
each time period. A technique had to have a frequency of at least 10 to be ranked. 
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